Financial Data Exploration
by Jiew Wan Tan
19th June 2016
knitr::opts_chunk$set(fig.width=12, fig.height=8, fig.path='Figs/', echo=FALSE,
warning=FALSE, message=FALSE)
This project explores the behavior and inter-relationships among different financial assets and indices. All data are time series data.
Financial and economic data are abundant and up-to-date. Correlations, causation and any form of relationships among indices, asset classes and instruments have always exist. As much as the complexity of the relationships are fascinating for data analysis, they are constantly in flux too as market condition change. It makes them challenging as well as irresistable to study. Financial time series are available from as short as 1 minute period. For uniformity, this exercise adopt daily time series and they are compiled into a csv file: dollar.csv, from 4th Jan 1995 to the second week of June 2016, a 20 years period. Since not all markets the assets trade in are open everyday (some markets open while the other are closed), we will see missing values among the datasets. In this case, they are omitted.
We are looking at dollar index, S&P500 index, gold, brent oil, treasury bonds at different maturities, LIBOR at different maturites, effective Fed fund rate, interest rate spreads and volatility index. Please note that treasury bond and T-bill are used interchangeably throughout this report, they actually mean the same instrument type.
Ther purpose in this exploration is the understand relationships. If the exploration found useful relationships, it can be used to generate actionable trading signal for algorithmic trading. For this reason, we are particularly interested in dollar Index and the Standard & Poor 500 index (S&P500) as there are easily a few popular instruments derived from these indices.
The function to calculate and display correlation coefficient on the plots:
1. Data loading, all data are cleaned and saved in a file: dollar.csv.
2. Convert date field to date format readable by R.
3. Setting the range of date to explore.
4. Summary of data.
## DATE dollar_index EFFR USD1MLIBOR
## 01/01/1997: 1 Min. : 89.03 Min. :0.040 Min. :0.1477
## 01/01/1998: 1 1st Qu.:100.63 1st Qu.:0.160 1st Qu.:0.2572
## 01/01/1999: 1 Median :108.10 Median :1.940 Median :2.3788
## 01/01/2001: 1 Mean :108.96 Mean :2.686 Mean :2.8416
## 01/01/2002: 1 3rd Qu.:116.60 3rd Qu.:5.240 3rd Qu.:5.3750
## 01/01/2003: 1 Max. :130.24 Max. :7.800 Max. :6.8213
## (Other) :5562 NA's :197 NA's :155
## USD3MLIBOR USD6MLIBOR USD12MLIBOR USD1MTCMR
## Min. :0.2228 Min. :0.3194 Min. :0.5335 Min. :0.000
## 1st Qu.:0.4416 1st Qu.:0.7137 1st Qu.:1.0547 1st Qu.:0.050
## Median :2.4300 Median :2.5838 Median :2.7588 Median :0.240
## Mean :2.9438 Mean :3.0639 Mean :3.2724 Mean :1.274
## 3rd Qu.:5.4162 3rd Qu.:5.4475 3rd Qu.:5.4588 3rd Qu.:1.860
## Max. :6.8688 Max. :7.1087 Max. :7.8125 Max. :5.270
## NA's :155 NA's :155 NA's :155 NA's :1849
## USD3MTCMR USD6MTCMR USD1YTCMR USD2YTCMR
## Min. :0.000 Min. :0.020 Min. :0.080 Min. :0.16
## 1st Qu.:0.120 1st Qu.:0.190 1st Qu.:0.330 1st Qu.:0.74
## Median :1.750 Median :1.920 Median :2.225 Median :2.75
## Mean :2.512 Mean :2.634 Mean :2.744 Mean :3.01
## 3rd Qu.:5.020 3rd Qu.:5.120 3rd Qu.:5.060 3rd Qu.:5.10
## Max. :6.420 Max. :6.670 Max. :7.320 Max. :7.68
## NA's :200 NA's :200 NA's :200 NA's :200
## USD3YTCMR USD5YTCMR USD7YTCMR USD10YTCMR
## Min. :0.280 Min. :0.560 Min. :0.910 Min. :1.430
## 1st Qu.:1.070 1st Qu.:1.730 1st Qu.:2.290 1st Qu.:2.890
## Median :3.115 Median :3.640 Median :3.950 Median :4.260
## Mean :3.223 Mean :3.626 Mean :3.959 Mean :4.221
## 3rd Qu.:5.120 3rd Qu.:5.150 3rd Qu.:5.340 3rd Qu.:5.372
## Max. :7.840 Max. :7.900 Max. :7.920 Max. :7.890
## NA's :200 NA's :200 NA's :200 NA's :200
## TED_Spread hy_bondspread BRENTOIL gold
## Min. :0.0900 Min. : 2.410 Min. : 9.10 Min. : 252.8
## 1st Qu.:0.2200 1st Qu.: 3.870 1st Qu.: 23.66 1st Qu.: 325.8
## Median :0.4000 Median : 5.250 Median : 44.98 Median : 459.5
## Mean :0.4958 Mean : 5.819 Mean : 54.11 Mean : 733.6
## 3rd Qu.:0.6100 3rd Qu.: 7.040 3rd Qu.: 78.64 3rd Qu.:1179.8
## Max. :4.5800 Max. :21.820 Max. :143.95 Max. :1895.0
## NA's :311 NA's :516 NA's :138 NA's :191
## VIX SP500
## Min. : 9.89 Min. : 460.3
## 1st Qu.:14.51 1st Qu.:1019.0
## Median :19.00 Median :1219.3
## Mean :20.51 Mean :1244.5
## 3rd Qu.:24.06 3rd Qu.:1414.0
## Max. :80.86 Max. :2130.8
## NA's :171 NA's :168## DATE dollar_index EFFR USD1MLIBOR USD3MLIBOR USD6MLIBOR
## 1 01/04/1995 94.3497 6.59 5.93750 6.50000 7.0000
## 2 01/05/1995 94.0717 5.72 5.89844 6.43750 6.9375
## 3 01/06/1995 94.3577 5.52 5.87500 6.43750 6.9375
## 4 01/09/1995 NA 5.52 5.87500 6.37500 6.8750
## 5 01/10/1995 94.5032 5.44 5.87500 6.37500 6.8750
## 6 01/11/1995 94.3459 5.44 5.87500 6.33594 6.8750
## USD12MLIBOR USD1MTCMR USD3MTCMR USD6MTCMR USD1YTCMR USD2YTCMR USD3YTCMR
## 1 7.8125 NA 5.85 6.57 7.15 7.62 7.75
## 2 7.6875 NA 5.88 6.61 7.32 7.66 7.83
## 3 7.6875 NA 5.90 6.61 7.26 7.64 7.81
## 4 7.6875 NA 6.00 6.67 7.27 7.68 7.84
## 5 7.6875 NA 5.91 6.62 7.21 7.63 7.78
## 6 7.6250 NA 5.79 6.51 7.13 7.58 7.73
## USD5YTCMR USD7YTCMR USD10YTCMR TED_Spread hy_bondspread BRENTOIL gold
## 1 7.81 7.82 7.82 0.81 NA 15.93 375.6
## 2 7.87 7.89 7.88 0.72 NA 16.13 375.3
## 3 7.87 7.89 7.87 0.70 NA 16.35 375.2
## 4 7.90 7.92 7.89 0.63 NA 16.08 372.4
## 5 7.85 7.87 7.84 0.64 NA 16.00 374.1
## 6 7.80 7.82 7.79 0.71 NA 16.40 375.6
## VIX SP500
## 1 13.53 460.71
## 2 13.50 460.34
## 3 13.13 460.68
## 4 13.33 460.83
## 5 12.52 461.68
## 6 12.15 461.66
Distribution & Time Series Characteristics for Dollar Index
The dollar index has been ranging between 80 to 140 over the past twenty year. Now hovering around 121, it is on the higher end of the range. From the historgram, it has a weak binomial distribution. So it is commonly seen to be ranging around 100 and 120. Its uptrend pathway is generally steeper than downtrend. The last 8 years has seen 2 uptrends that lasted 2-3 years, whereas the the uptrend before that took about 7 years. The 3 uptrends happened during the late nineties, the late 2000s and the early to mid 2010s.
In all following historgrams, means are marked with red lines, 25% and 75% quantiles are marked with yellow lines.
Distribution & Time Series Characteristics for S&P500
The S&P500 index has a normal distribution. We can observe that at some point of time, the dollar index correlates with the S&P500. In most other time, it propably relates more to other factors. The attraction of capital to the US capital market, part of it contributed by the stock markets (which the S&P500 tracks) should also contributes to USD demand and hence its uptrend. Though, this correlation could be hard to catch and both the dollar and stock markets are huge market with global participants’ interactions. There are just too many causations in their movements. The major causes are trade and capital flow, central banker’s manupulations. Naturally, when most participants are driving on the same cause at the same period, a trend is set for years.
Distribution & Time Series Characteristics for effective Fed fund rate
The EFFR has an unnatural, somewhat skewed distribution. We notice a large number of distribution concentrates in the 0.xx region. This is an unnatural cause of Federal reserve massive quantitative easing program that suppress interest rate at 0.25-0.5 since the subprime financial crisis in 2008. However, it is expected to move gradually.
Distribution & Time Series Characteristics for Gold
Gold is considered a commodity as well as a safe haven asset. It has enjoyed a long period of uptrend when interest rates have been set to exceptionally low. Sometime in early 2013, market has expectation for Fed to raise interest rate, as such the begin of gold downtrend. In the recent months, despite the backdrop of Fed’s rate rise, negative interest rates are seen in major economies such as Europe and Japan while emerging markets growth weakens. Gold has seen revival due to its safe asset status. So we can assume that gold has a negative correlation with interest rates and a positive correlation with risk sentiment.
Distribution & Time Series Characteristics for Brent Crude oil
Oil is a commodity asset. What contributes to its price are more on political, economical and environmental factors. Its histogram is somewhat skewed. the past ten years has been extremely volatile for oil.
Distribution Characteristics for VIX, High yield bond spread and TED Spread
VIX is an Index, TED spread is the difference between interbank loan interest rate and short term T-bill rate, High yield spread is the rates difference between risky and low risk bonds. All can be used as gauge for risk sentiment. The skewed histograms shows that most of the time , market sentiment is calm with occassional risk aversion sentiment.
Distribution Characteristics for LIBORs
From both LIBORs and Treasury bonds’ histograms, we observed that the longer maturity bonds are less skewed and more normal. The shorter term bonds’ histograms look more similar to effective Fed fund rate’s histogram (see above). We can attribute this to the power of Federal Reserve’s market manupulation on short term interest rate, with its ultimate goal to encourage bank lending to stimulate economy. The longer term bonds are less affected by it and more by market force.
Distribution Characteristics for Treasure Bonds
Bivariate exploration of various features: Scatter plot
We are experimenting S&P 500 and Dollar index’s relationship with other features, to see if we can find any interesting correlation. As for treasury bond and LIBOR, we select a few commonly use maturities from each set.
All scatter plots will be fitted with linear regression or loess lines with confidence internals. When a linear association is observed, a linear regression line is plotted. If the scatter plot shows a non-linear and more complicated relationship, a loess line will be plotted instead. Correlation coefficients will also be included in all plots. The complex relationships observed in most plots reflects that the plots deserve further investigation in a different study.
Bivariate exploration of interest rate spreads: Scatter plot
Bivariate exploration of gold: Scatter plot
Gold and oil, in contrast have strong negative correlation with most other assets, espciallly interest rate. Gold is viewed a safe haven in the market, during the time of market stress, together with treasury bonds, gold price will be bid up by investor rushing for safety. When bond price is up, causing its yield to drop (a fixed interest rate bond’s yield will be less attractive when it price goes up), as such gold price is highy correlated with bond price and has strong negative correlation with bond yield. Before we proceed further to investigate risk and interest rate-related assets , let’s look at gold price behavior with 3 assets:
Correlation Matrix: An Overview
Since we have many features, a correlation matrix will be handy. We notice that the two features: S&P 500 and dollar index we are studying here do not have obvious correlationship with most other assets. Particularly dollar, that imply its complexity with other assets in the markets.
VIX in Dollar Index and S&P500 distribution: Histogram
Volatity index (VIX) is known to be the fear index. When market is fearful, the VIX reflects that. VIX is currently moving between mid to late teens, not particularly fearful. It is when it has gone above 30. From the histograms below, we can tell most of the time the markets are not fearful. Though, as we see from S&P0500’s histogram, markets are fearful when they are at the lower end: < 1000 and between 105 to 113 for dollar index.
VIX in Dollar Index and S&P500 distribution: Scatter plot
The scatterplots show Dollar index & S&P500 relationships with VIX. There are weak positive (dollar) and weak negative correlations (S&P500).
Dollar Index & S&P500 behavior in risk sentiment
TED spread is the difference between the interest rates on interbank loans and on short-term U.S. government debt (“T-bills”). It can be used as a gauge of risk sentiments. (See note) From the scatterplots, we notice that when VIX is high, TED spread tends to be high too.
Note: TED is an acronym formed from T-Bill and ED (Eurodollar, LIBOR), the ticker symbol for the Eurodollar futures contract. TED spread is the difference between the three-month LIBOR and the three-month T-bill interest rate. The TED spread is an indicator of perceived credit risk in the general economy,since T-bills are considered risk-free while LIBOR reflects the credit risk of lending to commercial banks. An increase in the TED spread is a sign that lenders believe the risk of default on interbank loans (also known as counterparty risk) is increasing. Interbank lenders, therefore, demand a higher rate of interest, or accept lower returns on safe investments such as T-bills. When the risk of bank defaults is considered to be decreasing, the TED spread decreases.
Correlations of risk sentiment indicators
We know that risk sentiments plays a significant part in market behavior, the analysis thus far has shown how it is reflected in different asset class. We progress further by adding a variable into the bivariate plots, making multi-variate analysis. First, by reaffirming it with three risk indicator behave with each other. See notes about high yield bond spread: https://research.stlouisfed.org/fred2/series/BAMLH0A0HYM2 for further info about high yield bond spread.
We notice from all three scatter plots that VIX, TED Spread & high yield bond spread are very much correlated. Their indiviual line charts also show that they spiked up around the same period, or better say they response to market sentiment similarly.
More on other asset classes behavior with VIX
We notice that dollar has negative correlationships with gold and oil. That makes sense as both are denominated by dollar. When dollar goes up, they have to go down. However, there seems to be more between Dollar and gold.
Dollar and effective Fed fund rate (EFFR) has mild correlation as EFFR is somewhat manipulated by Federal Reserves (the United States’ central bank).
S&P500 is inversely correlated to High yield bond spread, this explains the stock market is down when risk sentiment is high, VIX color in the plot reaffirms that too.
The dollar index has a non linear association with high yield spread, which probably worth looking further into an equation that explains their relationship. VIX’s behavior in them is mixed.
The dollar index and S&P 500 looks to have a very complicated relationship. It will take some quantative analysis work to derive a method to explain it.
Final Plots: Characteristics of fixed income assets (bonds) of different maturities
The plots reveals that interest rate, their spread among short and long term maturities plays a part in risk sentiment. They have also shown coherent and consistent behavior from the plots above. Given market is very much affected by risk sentiment, we should see if these can help predict movement of S&P500 and the dollar.
Having looked at how risk sentiment plays out in different asset classes, we now look into how it is reflected in treasuries and LIBORs of varied maturities or better how bonds and LIBORs reflect that. Treasury bonds are viewed as a safe asset (risk free assets) in the market. When risk sentiment is high or imminent, the bonds market is likely the earliest to reflect that as investors rush for safe havens. When demand bids up bond price, naturally, their yields drop. LIBORs are interbank rates: borrowing between banks. When interbank borrowing market is stressed, the banks wil bid up LIBOR rates. We can observe form the boxplot that longer maturity bonds tend to have higher yields, and that shorter maturity bonds tend to be more volatile. The one month treasury bond even has a bunch of outliers.
Final Plot 1: Behavior of LIBOR and Treasury of the same maturity
Following the findings on risk, rates and spread, we sum it up by looking at TED Spread in action: how LIBOR and Treasury bond of the same matuirties relate to each other and together their response to risk sentiment.
We observe that LIBORs and treasury bonds are very correlated at all 4 shared maturities, when their rates go to the upper end, market tends to be in above average stress, moderately high risk sentiment. It is the extreme stressful market condition that distort their correlationship, LIBOR rates went high while treasury bonds stick to their lowest range, particularly the shorter maturity bonds. This very much explains the characteristic difference between LIBOR and treasury markets mentioned above.
This is an important finding so far in this exploration. When trader see a tendency of bond rates of the same maturity diversify from each other and both rates are not at their high end, the trader should take note. Though this finding has to be complimented by another perspective: time. Because it is enough to know the correlation and distribution. We should know who is leading and who is lagging to have an actionable signal in trading.
So far we have seen risk sentiments expressed in VIX, TED spread and high yield spread. In short, the spreads between risk-free and riskier assets. What about the spread between short term and long term asset of the same asset class, such a 1 year T-bill with a 10 year T-bill? In investment term, this is usually expressed in yield curves. While it is beyond the scope of his project to express yield curve over the years, I found a fanstastic 3D presentation of yield curve from 1990 to 2015 in New York Times:
http://www.nytimes.com/interactive/2015/03/19/upshot/3d-yield-curve-economic-growth.html?_r=1
When time comes into play, it has to do with market participants’ expectation from now to the term of the bond’s maturity. What does market expect from now until a year later when one year treasury bond expires? High inflation? Deflation? Better economic growth? recession? It should be reflected on the yield curve.
For now, the time series plot of all maturities of Treasury bond reflects the contraction and expansion of short and long term bond yield spread:
Treasury bonds from 1995- Present
An overview of how treasury bonds trend over the years.
LIBORS from 1995-Present
An overview of how LIBORs trend over the years.
LIBOR-T- bill spreads from 1995-Present
An overview of how LIBORs and treasury bills spreads (LIROR rate - treasury bill rate) trend over the years. The 3 month spread is actual TED spread. This plot is the time series plot for Final Plot 1: Behavior of LIBOR and Treasury of the same maturity.
Behavior of Treasury bonds vs VIX
The 1995-present time series capture the two major financial crisis: the internet bubble in the late nineties - early 2000s and the 2008 subprime financial crisis. We can see that T-bills started to behave unusually(the spreads between short and long term yield became very narrow) much earlier than before the market became panic (when VIX spikes up).
Intermarket behavior of S&P500 VS VIX
From the last two major financial crisis, VIX shot up to one of the highest when the stock market fell to almost the lowest. Most of the time, market rebounded before the panic is over. For other smaller market downtrends, or corrections, VIX also spiked up when market has dropped to its lowest point and was about to rebound. As such VIX as a fear index in risk sentiment is a lagging indicator.
What about VIX’s two other sentimental friends, TED Spread and High yield bond spread?
Final Plot 2: Intermarket behavior of TED spread and High yield bond spread vs S&P500
It seems TED and the other LIBOR-T-bill spreads and High yield bond spread did not react at the same time. High yield bond spread is almost as slow as VIX. Furthermore, high yield bond spread is quite volatile most of the time and as such not very useful as an indicator. The reason being is, it would be hard to tell which actions are for real and which is not, without the benefit of hindsight.
LIBOR-T-bill and TED spreads are the calmer beasts and have the wisdom to signal earlier. However, almost half of the stock market actions are in motion when TED spread reacted in a big way. Though, we do see a bit of TED spread jitters prior to that.
Now we shall revisit T-bills. From the plots above, if treasury bonds reacted much earlier than VIX, what about comparing it to the stock market?
Final Plot 3: Intermarket behavior of Treasury bonds vs S&P500
From here it is obvious that the contraction of treasury bond spread precedes S&P500 down trends. When the spread was in widest range, market trends up for years.
In both 2000-2001 and 2008 crisis, short term yield (the orange-pinkish lines) actually exceed long term yield (the blueish lines) before the crashes. It seems market expect inflation and economic growth in the longer terms to be gloomier than now! Or worse, a crash or crisis is imminent!
So yield spread (short term Vs long term maturity spread) works conversely with bond spread (high yield/riskier vs low yield/low risk). When yield spread is high, market is optimistic of the future, when yield spread is low, market is more pessimistic and risk-averse. While high yield bond or TED spread work the other way. The exploration so far has clarified types of yield and spread and how they behave under risk-on/risk-off situation. The first two final plots conclude that.
With this observation in mind, we can conclude that interest rate and their spread serve as a reliable indicator to signal of trouble. Looking forward, we can see that treasury bond spread is showing sign of contraction now, is a market downtrend imminent and is this the time to short the S&P500?
Intermarket behavior of Treasury bonds VS Dollar Index
Finally, since treasury bills are such a reliable indicator, we shall look at how it fare with dollar index. Unlike the earlier plots, this plot shows little relationship between bonds and dollar. As such the limitation of the technique we have used so far. It is meant to open room for further investigation beyond this project. More advanced techniques (to be discussed in Reflection) to be explored for the Dollar Index relationship with other assets.
Hypothetically, we can assume that the dollar plays many roles as an international reserve and trading currency in the global market, there are much more interactions in the works than the US stock market. Understandably, it would be challenging to find a trading clue from just a few visualizations.
Nonetheless we should still explore dollar correlation with treasury bonds. The result shows that they generally have weak positive correlations:
## Correlation of Dollar Index with 1 month Treasury bond = 0.1476853## Correlation of Dollar Index with 3 month Treasury bond = 0.06161503## Correlation of Dollar Index with 6 month Treasury bond = 0.05876886## Correlation of Dollar Index with 1 year Treasury bond = 0.06255479## Correlation of Dollar Index with 2 year Treasury bond = 0.08920609## Correlation of Dollar Index with 3 year Treasury bond = 0.1036078## Correlation of Dollar Index with 5 year Treasury bond = 0.1120118## Correlation of Dollar Index with 7 year Treasury bond = 0.1118852## Correlation of Dollar Index with 10 year Treasury bond = 0.07161541
Reflection
One of the challenges is to identify a list of variables (datasets) for the analysis, making sure they share the same frequency (monthly, weekly, daily, minute or hour). Without that it is pointless to do a time series analysis. Cleaning the data and making sure each dataset align in date is another labor-intensive task. In future development, I will make use of “quantmod”, an R package to extract data.
Another problem encountered with time series data is that, every dataset of the same time period has different length due to missing values (different trading days in different market), R’s correlation coefficient function does not handle datasets of different length, which make perfect makes sense. As such, instead of omitting missing values, we need to add NA in the dataset so that the same length retains. Fortunately, R’s ‘cor’ function has this feature, which is to add use=‘complete.obs’ to the function: cor(x, y, use=‘complete.obs’)
As for area plots in double Y-axis plots, the polygon function used to draw area plots doesn’t mix well with missing values, neither ommitting or adding ‘NA’ works. As such the missing values have to be filled with values, I have chosen to fill them with values between the values before and after the missing values by using this function interpNA(usd$dollar_index, method = “linear”). Neither mean nor median value work wells here as it will introduce a sudden change of value in the time series. The value before or after the missing value is probably not a good idea too as it could distort the mean, albeit slightly, in my opinion.
Once the datasets are identified, it takes some investigation to find out which one is useful and which is not. For example, effective Fed fund rates, although available daily and very up to date turns out is not as useful as it is manipulated by Fed, not completely driven by market force. Also, some asset class such as gold and oil’s behviors have too many factors (politics & environmental effect) in play that affect its price, so adopting them as a quantitative indicator can be confusing and risky at times. Cleveland Financial Stress indices at different flavors look promising but only available on a monthly basis.
There are probably more indicators required to build a robust predictive system or at least to get a reliable sense of market risk sentiments.
Compare to S&P500, the dollar has much more non-lienar, complex relationships with other assets. Not forgetting serial correlationship with itself and with other assets. Which means getting an actionable data-driven trading signal for the dollar will be very challenging. The depth and breath of study to investigate this into further detail will be beyond the scope of this project. I do realize that R offers many other statistical functions to help with analysis. Autocorrelation function (acf) and Cross-correlation (CCF) are among them.
Serial correlation is correlation among time lag(s) of the same dataset. (ACF is serial correlation with itself and CCF is with the other dataset). It is a form of time series analysis that can become useful if the right methods and parameters are fitted. Since it is not part of this course, it is not included.
This project shortlisted VIX, Treasury bonds, LIBORs, various interest rate spreads as reliable predictive indicators. These few instruments relationships are less complex and their behavior is more consistent over time. VIX is helpful but it is a lagging indicator, its value lies more on confirming risk sentiment. Nevertheless, we shouldn’t dismiss the other indicators such as gold, oil and others that are not included in this study. Their inte-relationships are not obvious in visuals but further investigation using other statistical methods as well as machine learning may yield useful predictions. Given the vast and steep learning curve, I’m humbly as well as enthusiastically looking forward to the machine learning course to extend the exploration.